fix(frontend): surface invocation errors instead of dumping HTML 404 pages by mmabrouk · Pull Request #4702 · Agenta-AI/agenta

mmabrouk · 2026-06-15T09:18:37Z

Context

When an LLM-judge evaluator ran against a misconfigured revision, its output was a blob of HTML instead of a score or an error.

Two cases caused this. A revision with no invocation URL (empty uri and url) resolved the request to the web app origin, which returned the app's 404 page. A revision whose service was unreachable returned an HTML error page instead of JSON. In both cases the execution path stored the raw HTML page as the run output, so the judge scored the page.

Changes

The execution paths now return a clear message instead of the HTML body.

Empty invocation URL: No invocation URL configured for this revision (empty uri and url).
HTML response body: Service unreachable at <url> (HTTP <status>).

The empty-URL message and the HTML detection used to be copy-pasted across four fetch paths. They now live in one dependency-free helper, @agenta/entities/shared/execution/invocationErrors, that every path calls:

executionRunner.ts and the playground web worker short-circuit an empty URL and detect HTML error bodies.
executionItems.ts keeps a resolved-empty URL empty instead of falling back to the web app origin, so the empty-URL guard can fire.
runnable/utils.ts uses the HTML detector in both the runnable and the built-in evaluator fetch paths.

The helper is a deep import (not the package barrel) so the web worker bundle does not pull in the rest of @agenta/entities.

Before, an HTML 404 surfaced as:

<!DOCTYPE html><html>...404: This page could not be found...</html>

After:

Service unreachable at https://.../test (HTTP 404)

JSON error bodies and plain-text errors still pass through unchanged.

Tests / notes

Added invocationErrors.test.ts covering HTML detection (doctype, leading tag, case and whitespace), the non-match cases (JSON, a tag later in the string, null/empty), and the message builders.
@agenta/playground unit suite passes (89 tests). @agenta/entities and @agenta/playground typecheck clean. pnpm lint-fix passes.
Verified both scenarios and the regression cases (JSON detail, plain text) against the real helper and the exact branch logic from each changed path. I did not click through the live stack: both running deployments bind-mount the main checkout, not this worktree, and a worktree redeploy was out of scope.

What to QA

Open the playground for a revision with no deployed service (empty invocation URL) and run it. The cell shows "No invocation URL configured for this revision", not an HTML page.
Run an evaluation where the app or evaluator service is down so the invoke returns an HTML 404. The output shows "Service unreachable at (HTTP 404)", and an LLM judge over it reports the error instead of scoring HTML.
Regression: run a healthy revision. Output renders as before. A real backend error with a JSON detail still shows that message.

…pages When a playground or evaluator revision had no invocation URL, or the invoke endpoint returned an HTML 404 page instead of JSON, the run recorded the raw HTML as its output. An LLM-judge evaluator then scored a blob of HTML instead of reporting the error. The execution paths now return clear messages: "No invocation URL configured for this revision..." when the URL is empty, and "Service unreachable at <url> (HTTP <status>)" when the body is HTML. Consolidates the empty-URL message and the HTML detection into one shared helper in @agenta/entities/shared/execution/invocationErrors so all four fetch paths share the same logic.

vercel · 2026-06-15T09:18:44Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	Jun 15, 2026 9:19am

coderabbitai · 2026-06-15T09:19:00Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1d13e46b-a87c-49bc-a638-ab41fd539422

📥 Commits

Reviewing files that changed from the base of the PR and between 56dc00e and 65e1852.

📒 Files selected for processing (8)

web/oss/src/components/Playground/hooks/useWebWorker/assets/playground.worker.ts
web/packages/agenta-entities/package.json
web/packages/agenta-entities/src/runnable/utils.ts
web/packages/agenta-entities/src/shared/execution/index.ts
web/packages/agenta-entities/src/shared/execution/invocationErrors.ts
web/packages/agenta-entities/tests/unit/invocationErrors.test.ts
web/packages/agenta-playground/src/state/execution/executionItems.ts
web/packages/agenta-playground/src/state/execution/executionRunner.ts

📝 Walkthrough

Summary by CodeRabbit

Release Notes

Bug Fixes
- Improved error messages when services are unreachable or misconfigured
- Added validation for missing invocation configuration
- Enhanced HTTP error response handling with better diagnostics
- More readable error reporting across execution workflows
Tests
- Added unit tests for invocation error reporting utilities

Walkthrough

A new shared invocationErrors.ts module is added to agenta-entities exporting MISSING_INVOCATION_URL_ERROR, isHtmlBody, and describeUnreachableService. These helpers are registered as a package subpath export and re-exported from the shared execution barrel. Three execution paths (executionRunner, playground.worker, runnable/utils) are updated to short-circuit on a missing invocation URL and to convert HTML error bodies into a readable "service unreachable" message. resolveInvocationUrl and buildExecutionItem in executionItems are also adjusted to return an empty URL when runtimePrefix is absent and to guard query-param appending accordingly.

Changes

Invocation error helpers and propagation

Layer / File(s)	Summary
New invocationErrors module, package export, and unit tests `web/packages/agenta-entities/src/shared/execution/invocationErrors.ts`, `web/packages/agenta-entities/package.json`, `web/packages/agenta-entities/src/shared/execution/index.ts`, `web/packages/agenta-entities/tests/unit/invocationErrors.test.ts`	Creates `invocationErrors.ts` with `MISSING_INVOCATION_URL_ERROR`, `isHtmlBody`, and `describeUnreachableService`; registers the `./shared/execution/invocationErrors` subpath export; re-exports from the barrel; adds a Vitest suite covering all three.
Invocation URL resolution and query-param building `web/packages/agenta-playground/src/state/execution/executionItems.ts`	`resolveInvocationUrl` returns `""` when `runtimePrefix` is absent; `buildExecutionItem` resolves the URL first and skips query-param appending when the result is empty.
HTML body and missing-URL error handling across execution paths `web/packages/agenta-playground/src/state/execution/executionRunner.ts`, `web/oss/src/components/Playground/hooks/useWebWorker/assets/playground.worker.ts`, `web/packages/agenta-entities/src/runnable/utils.ts`	Imports the new helpers; adds an early-return `"error"` result when `invocationUrl` is falsy; detects HTML response bodies on non-2xx responses and replaces them with `describeUnreachableService` output in all three callers.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: fixing frontend invocation errors by surfacing readable messages instead of dumping raw HTML 404 pages.
Description check	✅ Passed	The description is detailed and directly related to the changeset, explaining the problem, solution, affected files, and testing approach.
Docstring Coverage	✅ Passed	Docstring coverage is 71.43% which is sufficient. The required threshold is 60.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/playground-invocation-error-handling

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-15T09:31:19Z

Railway Preview Environment


Preview URL	https://gateway-production-b54d.up.railway.app/w
Project	`agenta-oss-pr-4702`
Image tag	`pr-4702-d6503c7`
Status	Deployed
Railway logs	Open logs
Workflow logs	View workflow run
Updated at 2026-06-15T09:31:18.466Z

dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jun 15, 2026

dosubot Bot added the Frontend label Jun 15, 2026

vercel Bot deployed to Preview June 15, 2026 09:19 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(frontend): surface invocation errors instead of dumping HTML 404 pages#4702

fix(frontend): surface invocation errors instead of dumping HTML 404 pages#4702
mmabrouk wants to merge 1 commit into
mainfrom
fix/playground-invocation-error-handling

mmabrouk commented Jun 15, 2026

Uh oh!

vercel Bot commented Jun 15, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jun 15, 2026 •

edited

Loading

Summary by CodeRabbit

Release Notes

Walkthrough

Changes

Estimated code review effort

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mmabrouk commented Jun 15, 2026

Context

Changes

Tests / notes

What to QA

Uh oh!

vercel Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Release Notes

Walkthrough

Changes

Estimated code review effort

Uh oh!

github-actions Bot commented Jun 15, 2026

Railway Preview Environment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Jun 15, 2026 •

edited

Loading

coderabbitai Bot commented Jun 15, 2026 •

edited

Loading